AO-A043  079 


UNCLASSIFIED 


I 


END 


9-77 


AlR  FORCE  HUMAN  RESOURCES  LAB  BROOKS  AFB  TEX 
FAMILIARITY  EFFECTS  ON  TASK  DIFFICULTY  RATINOS.(U) 
JUN  77  K 6 KOYM 


AFhRL-TR-77-25 


AFHRL-TR-77-25 


AIR  FORCE  B 


/ I o 


FAMILIARITY  EFFECTS  ON  TASK 

DIFFICULTY  RATINGS  T 


By 

Kenneth  G.  Koym 


OCCUPATION  AND  MANPOWER  RESEARCH  DIVISION 
Lackland  Air  Force  Base,  Texas  78236 


June  1977 

Interim  Report  for  Period  1 July  1973  — 28  February  1977 


Approved  for  pubUc  release;  distribution  unlimited. 


AUG  1977 


. < A 


LABORATORY 


AIR  FORCE  SYSTEMS  COMMAND 

BROOKS  AIR  FORCE  BASE, TEXAS  78235 


NOTICE 


When  US  Government  drawings,  specifications,  or  other  data  are  used 
for  any  purpose  other  than  a definitely  related  Government 
procurement  operation,  the  Government  thereby  incurs  no 
responsibility  nor  any  obligation  whatsoever,  and  the  fact  that  the 
Government  may  have  fomiulated,  furnished,  or  jn  any  way  supplied 
the  said  drawings,  specifications,  or  other  data  is  not  to  be  regarded  by 
implicauon  or  otlierwise,  as  in  any  manner  licensing  the  holder  or  any 
other  person  or  corporation,  or  conveying  any  rights  or  permission  to 
manufacture,  use,  or  sell  any  patented  invention  that  may  in  any  way 
be  related  thereto. 

This  interim  report  was  submitted  by  Odcupation  and  Manpower 
Research  Division.  Air  Force  Human  Resources  Laboratory,  Lackland 
Air  Force  Base,  Texas  78236,  under  project  7734^  with  HQ  Air  Force 
Human  Resources  Laboratory  (AFSC),  Brooks  Air  Force  Base,  Texas 
78235. 

This  report  has  been  reviewed  and  cleared  for  open  publication  and/or 
public  release  by  the  appropriate  Office  of  Infomtation  (Of)  in 
accordance  with  AFR  190-17  and  DoDD  5230.9.  There  is  no  objection 
to  unlimited  distribution  of  this  report  to  the  public  at  large,  or  by 
DDC  to  the  National  Technical  Information  Service  (NTIS). 

This  technical  report  has  been  reviewed  and  is  approved  for  publication. 

RAYMOND  E.  CHRISTAL,  Technical  Director 
Occupation  and  Manpower  Research  Division 


DAN  D.  FULGHAM,  Colonel,  USAF 
Commander 


Unclassified 

security  classification  of  this  pace  ^Wh.n  Dmtm  Enler.rfj 


REPORT  DOCUMENTATION  PAGE 


READ  INSTRUCTIONS 
BEFORE  COMPLETING  FORM 


Y BCLPQP  T a" 

/ AFHRLTR-77-25  ' / 


,2  GOVT  ACCESSION  NO  ClPlEN'»”S  CATA'^OG  number 


f Mtrmr 


FAMILIARITY  EFFECTS  ON  TASK  DIFFICULTY  RATINGS 
jt  r"  ^ 


‘tvml  I/T-Btnygr  it  BPBIgn  ?B-.'FPTt: 
Interim  y-  ^ ' 


lJuly»73  28  I'l 


91Z. 


Ja — autmortai 


i/o, 


Kenneth  G 


i^oym  I 


I Piwr 'JHtilN  U UWU  REPOP’’  NuM^r 


sr  UWU  RE 

mTract  on  GRAfr” 


9 PERFORMING  organization  NAME  AND  ADDRESS 

Occupation  and  Manpower  Research  Division 
Air  Force  Human  Resources  Laboratory  ^ 
Lackland  Air  Force  Base,  Texas  78235 


11.  CONTROLLING  OFFICE  NAME  AND  ADDRESS 

HO  Air  Force  Human  Resources  Laboratory  (AFSC) 
Brooks  Air  Force  Base,  Texas  78235 


14  monitoring  agency  name  ft  ADDRESSri/  different  from  Controlling  Office) 


i 

/ • I 


10  program  element  PROJFC.''  T AS*- 

AREA  ft  #»ORK  UNIT  NJMBtWS 

527031 

77340701 


II  REPORT  Dll' 

Jun«  P>77 


z 


n NUMBER  OF  PAGES 

i: 


15  security  CLASS,  fof  this  r».por*. 

Unclassified 


15*  DECLASSIFICATION  DOWNGRADING 
SCHEDULE 


16.  distribution  ST  ATEMENT  ro/  fh/s  RepofO 


Approved  for  public  release;  distribution  unlimited. 


17.  DISTRIBUTION  STATEMENT  (of  the  absfracf  onfered  In  Block  20,  if  different  from  Report) 


18-  Supplementary  notes 


19.  KEY  WORDS  (Continue  on  reverse  side  if  necessery  and  identify  by  block  number) 

task  factors 
task  difficulty 

task  familiarity  , y4'|Ly 


interrater  reliabQity  { r'  r lO 


'.  scaling  context  effects 


JTRACT  (Continue  on  ravefae  side  If  necessary  end  Identify  by  block  number) 

^This  report  examines  interrater  reliability  estimates  for  task  difficulty  raters  having  differing  levels  of 
familiarity  with  rated  tasks.  Data  were  collected  from  455  NCO  supervisors  who  rated  the  difficulty  and  familiarity 
of  424  tasks  in  the  Aircraft  Electrical  Repair  career  ladder  task  inventory.  The  results  showed  that  tlie  inlcrrater 
reliability  estimates  (R););)  decreased  from  .930  to  .802  for  six  task  difficulty  rating  scale  conditions  in  which  ratings 
were  eliminated  due  to  levels  of  familiarity.  This  finding  suggests  that  little  is  to  be  gained  from  cliiinnaiing  task 
difficulty  ratings  based  upon  an  experienced  judge’s  level  of  familiarity  with  tasks. 


A 


DD  I 1473  EDITION  OF  I NOV  65  IS  OBSOLETE 


■yev 


Unclassified 


security  CLASSIFICATION  OF  THIS  PAGE  'H*hen  Defe  F.ntered' 


PREFACE 


This  research  was  initiated  under  project  ll'iA,  Occupational  and  Career 
Management  Research;  * task  773402,  Development  and  Appraisal  of  Methods  for  Job 
Evaluation.  The  analyses  were  completed  under  task  773407,  Development  and 
Assessment  of  Methods  for  Determining  the  Reqairements  of  Air  Force  Jobs;  work  unit 
77340701,  Development  of  Methods  for  Specifying  Education,  Training  Aptitude,  and 
Experience  Requirements  for  Air  Force  Jobs. 

Technical  assistance  and  suggestions  have  been  made  by  Dr.  Raymond  E.  Christal, 
Squadron  Leader  Kenneth  Goody,  Dr.  Joe  Hazel,  Mr.  William  J.  Phalen,  and  Dr. 
Llewellyn  N.  Wiley.  The  draft  manuscript  was  typed  by  Mrs.  Helen  Widner  and  proofed 
by  Mrs.  Joyce  Glorgia. 


^ /VC’T- 

^ar/K/f 

V 


TABLE  OF  CONTENTS 


P»»e 


I.  Introduction  5 

II.  Method 5 

Data  Collection 5 

Data  Analysis  Design  5 

IIL  Results  and  Conclusions 6 

Interrater  Refiabdity  Estimates 6 

Conclusions  and  Implications 7 

References 8 

Appendix  A;  Instructions 9 

Factor  I — Task  Difficulty 9 

Factor  II  — Task  Familiarity 10 


3 


A/ or 


familiarity  effects  on  task  difficulty  ratings 


L INTRODUCTION 

Work  difficulty  can  be  defined  in  several  ways 
(Madden,  1962).  Mead  and  Chiistal  (1970)  found 
task  difficulty  defined  as  “the  time  needed  to 
learn  to  perform  a task  satisfactorily”  to  be  of 
great  benefit  in  predicting  supervisors’  job 
difficulty  ranking  policies.  Equations  have  been 
developed  which  compare  jobs  within  a career 
ladder  in  terms  of  this  kind  of  difficulty  (Koym, 
1977  ; Mead  & Christal,  1970). 

Many  applications  have  been  made  of  task 
difficulty  rating  data.  It  has  been  possible  to 
compare  the  “difficulties”  of  different  career 
ladders  using  task  difficulty  benchmark  scales 
(Fugill,  1971).  Carpenter  and  Christal  (1972)  used 
an  average  task  difficulty  per  unit  time  spent 
variable  to  predict  the  grade  level  of  5,485  civilian 
jobs.  Goo(fy  and  Watson  (1975)  also  used  this 
variable  to  predict  task  training  priorities. 

Improvements  have  been  made  in  the  reliability 
of  rating  data  by  eliminating  ratings  Irom  widely 
divergent  raters  (having  a statistically  significant 
difference  from  other  raters)  and  raters  whose 
performance  appeared  to  be  in  violation  of  rating 
instructions.  An  application  of  this  technique  is 
illustrated  in  recent  reports  by  Goody  (1976)  and 
Koym  (1977).  However,  relatively  little 
experimental  work  has  been  reported  regarding  the 
influence  of  the  level  of  familiarity  on  rater 
behavior.  The  current  system  of  task  evaluation 
makes  the  implicit  assumption  that  the  level  of 
familiarity  possessed  does  not  affect  the  ratings 
provided  by  experienced  personnel;  yet,  no 
systematic  check  has  been  made  of  this 
assumption. 

This  study  investigates  the  effect  of  raters’ 
familiarity  on  difficulty  ratings  assigned  to  tasks. 
It  does  so  by  asking  the  job  incumbents  to  rate  the 
degree  of  familiarity  they  possess  regarding  each 
task  and  the  relative  amount  of  time  they  estimate 
to  be  needed  to  learn  to  perform  each  task.  The 
study  attempts  to  discover  if  there  is  a systematic 
relationship  between  the  levels  of  familiarity 
which  raters  possess  with  regard  to  a task  and  the 
level  of  difficulty  that  they  assign  to  that  task. 
Given  a specific  Air  Force  job  inventory,  what 
effect  does  familiarity  with  a task  have  on  a rater’s 
estimate  of  the  relative  difficulty  of  that  task? 


n.  METHOD 

The  Aircraft  Electrical  Repair  career  ladder 
(423X0),  for  which  a job  inventory  had  been 
constructed,  administered,  and  analyzed,  was 
selected  for  the  investigation.  The  inventory 
contained  424  tasks  of  which  a subset  was  believed 
to  be  relatively  unknown  to  a major  part  of  the 
survey  population.  From  the  outset,  task  difficulty 
was  recognized  to  be  a complex  affair  (Madden, 
1960,  1961,  1962;  Lecznar,  1971).  In  this  study, 
difficulty  was  understood  to  involve  a job-oriented 
judgment  about  the  properties  of  a t^,  whereas 
familiarity  was  understood  to  involve  a personal 
acquaintance  judgment  about  a task  (Madden, 
1960, 1962). 

Seven-point  relative  scales  were  used  to  obtain 
the  difficulty  and  familiarity  ratings.  Non- 
commissioned officers  (NCO)  were  asked  to 
perform  two  operations  sequentially:  (a)  to  rate 
each  task  for  difficulty  based  on  the  time  needed 
to  learn  to  do  the  task  satisfactorily  using  the  scale 
ranging  from  “very  much  below  average”  to  “very 
much  above  average,”  and  (b)  to  rate  each  task  for 
familiarity  based  on  the  level  of  acquaintance  or 
knowledge  possessed  about  what  is.  involved  in 
doing  the  task  using  the  scale  ranging  from  “none” 
to  “complete”  familiarity.  For  easy  rater  reference 
the  above  instructions  and  scales  were  printed  at 
the  top  of  each  page  of  the  task  inventory. 
Appendix  A contains  the  instructions  included  in 
the  survey  booklets. 

Data  Collection 

Data  collection  was  handled  by  mail  via 
consoUdated  base  personnel  offices.  Complete 
forms  were  returned  by  455  job  incumbents  who 
possessed  duty  Air  Force  specialty  codes  (DAFSC) 
at  the  7-  and  9-skill  levels. 

Data  Analysis  Design 

To  test  the  impact  of  task  familiarity  on  task 
difficulty  interrater  reliability,  a series  of  computa- 
tions was  required  to  establish  the  various  rating 
conditions  of  interest.  These  'rating  conditions 
were  obtained  in  a three-step  process  by:  first, 
identifying  task  difficulty  ratings  with  seven 
differing  degrees  of  rater  familiarity.  These  seven 
degrees  of  familiarity  were  represented  by  the 
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familiarity  scale  values  7,  6 through  7,  S through 
7,  4 through  7,  3 throu^  7,  2 throu^  7,  and  1 
through  7 (the  full  scale  range);  second, 
sequentially  setting  tasks  associated  with  each 
far^arity  rating  scale  condition  equal  to  1 and 
those  tasks  associated  with  all  other  scale  condi- 
tions equal  to  0;  third,  obtaining  rater-by-task 
cross-products  between  each  task  familiarity  rating 
condition  and  the  task  difficulty  ratings.  This 
produced  seven  continuous  task  difficulty  rating 
factors  scaled  in  terms  of  a specific  level  or  kind  of 
rater  familiarity. 

Each  of  the  task  difficulty  rating  conditions 
was  input  to  the  CODAP  REXALL  interrater 
reliability  routine  (Stacey,  Weissmuller,  Barton,  & 
Rogers,  1974)  which  applies  the  components  of 
variance  technique  to  obtain  reliability  estimates. 
The  sample  value  for  each  factor  was  stepped  up  in 
terms  of  the  average  (k)  number  of  raters  in  each 
rating  condition  using  the  Spearman-Brown 
prophecy  formula.  Hie  individual  difficulty  ratings 
were  adjusted  (for  differences  in  the  raters’  frames 


of  reference)  to  a mean  of  5.0  and  SD  of  1.0  by 
the  CO  DAP  REXALL  routine.  This  was  done  to 
remove  error  due  to  systematic  differences  in 
rating  tendencies  and  to  control  for  familiarity 
context  effects. 


m.  RESULTS  AND  CONCLUSIONS 

Interrater  Reliability  Estimates 

Table  1 reports  reliability  estimates  for  the  task 
difTiculty  ratings  obtained  for  various  levels  of 
rater  familiarity.  The  intraclass  correlation 
coefficients  (Rn)  indicate  higher  agreement  for 
raters  who  were  completely  familiar  (assigned 
values  of  7)  than  for  ratings  assigned  over  the  full- 
scale  range  (1  through  7).  Raters  who  claimed  a 
great  deal  or  complete  familiarity  with  particular 
tasks  had  higher  task  difficulty  interrater 
reliability  estimates  than  raters  who  claimed  very 
little  or  no  familiarity  with  the  tasks. 


Table  I.  Interrater  Reliability  Estimates 
for  Difficulty  Ratings  for  Different 
Levels  of  Familiarity* 


Ratins  Seala  CondlMora 


*Ratings  were  standardized  by  CODAP  REXALL;  N Raters  = 455. 
^FuU  scale  range. 

‘'Represents  one-fifth  of  the  actual  k number  of  raters. 


However,  the  decrement  in  the  Spearman- 
Brown  (R|,|[)  values  reported  for  the  different 
rating  scale  conditions  indicates  that  considerable 
loss  in  the  stability  of  the  mean  task  difficulty 
vectors  occurs  when  “nonfamiliar”  ratings  are 
eliminated.  The  additional  ratings  produce  higher 
agreement  (Rj^c)  in  the  mean  vector  as  is  noted  for 
rating  scale  condition  ( 1 through  7)  versus  rating 
scale  condition  (7). 

The  average  number  of  raters,  k,  represented  in 
Table  1 is  only  1/5  of  the  actual  number  of  raters 
available.  The  lesser  range  was  used  here  to  bring 
the  number  of  judges  into  a range  comparable  to 
the  number  of  raters  normally  available  in  Air 
Force  research  studies. 


As  noted  in  Table  1 , the  intraclass  correlation 
coefficients  for  the  task  difficulty  ratings  in  this 
career  ladder  were  not  particularly  hi^  and 
probably  should  be  explained.  This  may  be  a 
function  of  (a)  the  block  of  relatively  unknown 
tasks  which  had  been  recently  added  to  the  task 
inventory,  and  (b)  the  low  task  performance  rate 
present  in  this  ladder.  The  low  task  performance 
rate  was  determined  using  the  most  recent  job 
inventory  survey  previously  conducted  on  1581 
jobs  (Tartell,  1974).  In  li^t  of  this  explanation,  it 
may  be  of  interest  to  replicate  the  study  in  other 
career  ladders  where  higher  reliability  coefficients 
could  be  expected. 
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Conclusions  and  Implications 

The  results  in  this  study  were  based  upon 
ratings  made  by  NCO  supervisors  who  were  asked 
to  rate  all  424  tasks  in  the  job  inventory  survey  for 
(a)  task  difficulty,  and  (b)  task*  familiarity.  Task 
difficulty  rating  conditions  were  developed  for 
raters  expressing  various  levels  of  familiarity  with 
tasks  being  rated.  Interrater  reliability  estimates 
were  computed  for  these  rating  conditions. 
Although  the  intraclass  correlation  coefficients 
showed  that  the  rating  sets  became  more 
consistent  as  unfamiliar  ratings  were  eliminated, 
the  stability  of  the  mean  task  difficulty  vectors 
decreased  when  these  ratings  were  eliminated.  This 
increase  in  the  internal  consistency  of  the  ratings 
simply  does  not  overcome  the  loss  in  the  stability 
of  the  mean  vector. 


While  previous  research  on  familiarity  effects 
(Christal  & Madden,  1960;  Madden,  1960,  1961) 
' has  indicated  that  judges  should  have  wide  general 
experience  in  an  area  being  evaluated,  the  results 
for  this  study  have  indicated  that  a more  specific 
level  of  fam^arity  is  not  necessarily  required  to 
produce  reliable  jugmoits.  It  would  be  of  interest 
to  replicate  this  study  in  other  career  ladders 
\»dieie  higher  reliability  coefficients  exist  among 
raters. 

In  keeping  with  the  present  system  these  results 
appear  to  support  the  contention  that  non- 
commissioned officer  supervisors  should  be 
instructed  to  provide  task  difficulty  ratings  for  all 
the  tasks  in  job  inventory  surveys  being 
administered. 
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APPENDIX  A:  INSTRUCTIONS 


FACTOR  I - TASK  DIFFICULTY 


DO  NOT  CONSIDER  FACTOR  11  BEFORE  ALL  TASKS  HAVE  BEEN  RATED  ON  THIS  FACTOR  ! 

I 


Instructions 


STEP  1.  Develop  a frame  of  reference  for  rating  task  difficulty.  Do  this  by  scanning  the  entire  listing  of  | 

tasks.  Pick  out  some  easy  tasks  which  can  be  learned  in  a short  time,  some  difficult  tasks  can  be  learned  j 

only  after  considerable  length  of  time,  and  then  other  tasks  which  fall  between  these  two  extremes.  The  j 

tasks  which  fall  at  or  near  the  middle  of  the  range  should  then  be  used  as  your  reference  point  forjudging  j 

the  difficulty  of  all  tasks  in  the  inventory.  Use  this  reference  point  in  completing  STEP  2. 

STEP  2.  Estimate  the  time  needed  to  learn  to  do  each  task  satisfactorily  compared  with  other  tasks  in  the 

career  ladder.  Use  the  scale  shown  here  and  at  the  top  of  the  difficulty  column  on  each  page  to  rate  every  i 

task. 

1 . Very  much  below  average  j 

2.  Below  average 

3.  Slightly  below  average 

4.  About  average 

5.  Sightly  above  average 

6.  Above  average  j 

7.  Very  much  above  average  j 

Begin  with  the  first  task  in  the  booklet  and  give  eadi  task  a difficulty  rating  from  1 to  7;  record  the  value  J 

opposite  the  task  statement  in  the  column  titled  “DIFFICULT.”  Rate  every  task  on  each  page.  Remember  j 

(from  STEP  1)  that  you  are  comparing  each  task  with  the  other  tasks  in  the  career  ladder.  gj 

Record  your  best  estimate  of  difficulty  even  on  those  tasks  which  you  may  believe  are  no  longer  done  in  i 

your  career  ladder.  Turn  to  page  1 and  rate  every  task  in  this  booklet  for  difficulty  now.  j 
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FACTOR  II  - TASK  FAMILIARITY 


Instructions  cont’d 


STEP  3.  After  you  have  rated  all  tasks  for  task  difficulty,  go  back  to  the  first  task  in  the  booklet.  Then, 
estimate  the  degree  of  familiarity  which  you  have  with  each  task.  Consider  the  level  of  acquaintance 
(frequency  of  contact)  or  knowledge  you  possess  about  what  is  involved  in  doing  each  task.  Use  the  scale 
shown  here  and  at  the  top  of  the  familiarity  column  on  each  page. 

1 . None 

2.  Very  little 

3.  Some 

4.  Moderate 

5.  Considerable 

6.  Great  deal 

7.  Complete 

Begin  with  the  first  task  in  the  booklet  and  give  every  task  a familiarity  rating  from  1 to  7;  record  the  value 
opposite  the  task  statement  in  the  column  titled  “FAMILIARITY.” 

Rate  every  task  in  this  booklet.  Estimate  familiarity  even  on  those  tasks  wK  ch  you  may  believe  are  no 
longer  done. 

STEP  4.  Write  any  comment  you  would  care  to  make  regarding  the  task  familiarity  factor  or  the  task 
difficulty  factor,  or  your  ratings  on  either  factor,  on  the  blank  pages  following  the  last  task  statement.  Any 
information  which  might  increase  the  value  of  this  study  or  future  studies  of  this  type  will  be  appreciated. 

STEP  5.  Review  the  booklet  to  see  that  you  have  (a)  completed  all  items  of  background  information,  and 
(b)  given  each  task  statement  a legible  rating  between  1 and  7 on  both  the  task  difficulty  factor  and  the 
task  familiarity  factor.  Return  this  booklet  to  the  CBPO  for  mailing  to  the  AFHRL/PEOE,  Lackland  AFB, 
TX  78236. 
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